PCX 



WORLD INTOLLECTUAL PROPERTV ORGANEATON 
Inteniational Boreso 




INTERNATIONAL APPUCATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (fCT) 



(51) Internatioiial Patent Clasafication ^ : 




(11) Interaatkmal Publkattoii Nmnber: 


wo 98/09270 


G09G5AH) 


Al 


(43) Internationa] Publication Date: 


5 March 1998 (05.03.98) 



(21) International AppUcation Number: PCT/US97/14I76 

(22) International FHing Date: 28 August 1997 (28.08.97) 



(30) Priority Data: 
60/024,780 
60/028.028 
60/036.195 



28 August 1996 (28.08.96) US 
9 October 1996 (09.10.96) US 
21 Januaiy 1997(21.01.97) US 



(71) Applicant (for all designated States except US): VIA, INC. 

(US/US); 1 1 Bridge Square. Northfield. MN SS057 (US). 

(72) Inventors; and 

(75) Inventors^Applicants (for US only): CARROLL. David. W. 

HUS]; 11 Bridge Square. Northfield, MN SS0S7 (US). 

CARROLU James, L. (-AJS]; 1 1 Bridge Square. Northfield, 
i MN 55057 (US). CASE. Steven, V. [-/US]; 11 Bridge 

Square. Northfield. MN 55057 (US), 

(74) Agents: HIENZ. William. M., Ill et al.; Patterson & Kcough, 
P.A.. 1200 Rand Tower, 527 Marquette Avenue South. 
Minneapolis. MN 55402 (US). 



(81) Designated States: AL. AM, AT. AU. AZ. BA. BB. BG. BR. 
BY, CA, CH. CN. CU, CZ. DE. DK. EE. ES, FI. GB. GE, 
GH. HU, IL, IS. JP, KE. KG, KP. KR. KZ, LC, LK. LR, 
LS. LT, LU. LV, MD. MG. MK. MN. MW, MX, NO, NZ, 
PL, PT, RO, RU, SD, SE. SG. SI, SK, SL, TJ. TM, TR, 
TT, UA. UG, US, UZ, VN, YU, ZW, ARIPO patent (GH, 
KE, LS, MW, SD, SZ, UG, ZW), Eurasian patent (AM, AZ, 
BY, KG, KZ, MD, RU, TI, TM), European patent (AT, BE, 
CH, DE. DK, ES. FI, FR, GB, GR, IE. IT, LU, MC. NL, 
PT, SE), OAPl patent (BP, BJ. CF, CG, a. CM. GA. GN. 
ML, MR. NE. SN. TD. TG). 



Publisbed 

With international search report. 

Before the expiration of the time limit for amending the 
claims and to be republished in the event of the receipt of 
amendments. 



(54) Title: TOUCH SCREEN SYSTEMS AND METHODS 
(57) Abstract 

A screen peripheral system according to an embodiment of the invention 
includes a computing device for producing a main image and a touch-activated 
input device (105) for generating and displaying a composite image (1 10) visible 
to a user. The composite image (1 10) simultaneously includes a representation 
of at least one key. for example a QWERTY keyboard, for activating an input 
function, and the main image provided by the computing device. The keyboard 
representation (105) preferably is laid over the main image. According to one 
ibodiment, the main image is an output image generated by an qjplication 
iing executed by the computing device. Odier touch screen sy^ems and 
methods are also disclosed. 



l^ml 
Sbeii 




FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party (o the PCT on die front pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


Spain 


LS . 




SI 


Slovenia 


AM 


Anncnia 


n 


FmlaDd 


LT 




SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Lmesnbouig 


SN 


Senegal 


AU 


Australia 


CA 


Gabon 


LV 


Latvia 


sz 


Swaziland 


AZ 


Azertnijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


BA 


Bosnia and Henegovina 


GE 


Gcozgia 


MO 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


GH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 


GN 


Guinea 


MK 


The former Yugoslav 


TM 


Turionenisian 


BF 


BTuiina Faso 


GR 


Greece 




Republic of Macedonia 


TR 


Ttnkcy 


EG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BR 


Brazil 


IL 


bnel 


MR 


Maaritania 


UG 


Uganda 


BY 


Belann 


IS 


Iceland 


MW 


Malavi 


US 


United States of Amcika 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


uz 


Uzbekbtan 


CF 


Central Aincan Republic 


JP 




NE 


Kiger 


VN 


Viet Nam 


CG 


Congo 


KB 


Kenya 


NL 


Netheriands 


YU 


Yugoslavia 


CH 


Switzcriand 


KG 


Kyrgyzstan 


NO 


Norway 


ZW 


Zimbabwe 


CI 


Cfte d'lvoire 


KP 


Democraiic People's 


NZ 


New Zealand 






CM 


CancrooQ 




Rqmbllc of Korea 


PL 


Inland 








Quna 


KR 


Repoblic of Koiea 


PT 


Portugal 






CU 


Cuba 


KZ 


Karakitan 


RO 


Romania 






CI 


Czech Repobltc 


LC 


Sahn Uida 


RU 


Russian Federuioo 






D£ 


Oennany 


U 




SD 


Sudan 






DK 
EB 


DeDnurt 
Estonia 


LK 
LR 


Sri Lanka 
Liberia 


SB 
SG 


Sweden 
Siagapoic 







wo 98/09270 



PCr/US97/14176 



TOUCH SCREEN SYSTEMS AND METHODS 
CRQSS'l^fERENCE TO RSIATEP AFFUCATIONS 

5 The subject matter of this application is related to the subject matter 

of commonly assigned U.S. Provisional Applications Nos. 60/024,780, filed 
August 28, 1996, 60/028,028, filed October 9, 1996, and 60/036,195, filed 
January 21, 1997, all of which provisional applications are incorporated by 
reference herein and priority to which is claimed under 35 U.S.C. §119(e). 

10 

BACKGI^QUND OF THE JtNVENTIQN 
1. Field of the Invention 

The invention relates to touch-sensitive input and output 
devices, and more particularly, to touch screens that simultaneously can 

15 display a keyboard-type image superimposed on a background image. The 
invention also relates to other features of touch-sensitive i/o devices. 
Embodiments of the invention have particular application to wearable- 
computing devices and environments, although non-wearable 
embodiments are also contemplated. 

20 2. Description of Related Art 

It is known in the art to superimpose a keyboard over an 
image that is output by an application being executed on a computer, i.e. to 
form a "phantom" keyboard, on a touch-sensitive input display device. 
U.S. Patent No. 5,581,243 to Ouellette et al., for example, which is 

25 incorporated by reference herein in its entirety, describes a system for 
displaying a simulated keyboard on a touch-sensitive display, without 
occluding from view an underlying application output image. An output 
image is generated and displayed for a first period of time at a first 
luminous radiation intensity having a predetermined persistence. A 

30 simulated keyboard image is generated and displayed for a second, shorter 
period of time at a second, lower luminous radiation intensity having the 
predetermined persistence. The keyboard and output images are generated 
in alternation, and the amount of time each is displayed is controlled, to 
try to create the illusion that both images are being displayed continuously. 
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The system described in Ouellette et al., however, is believed 
to suffer a number of disadvantages. First, it is believed that flashing 
between application and keyboard screens, as the reference discloses, 
would significantly reduce the refresh rate achievable relative to a 

5 continuously displayed application output (or a keyboard) image. This 
reduction could cause considerable detriment to the viewer and 
potentially could confuse the displayed interpretive features. Second, it is 
believed that displaying the keyboard and the application output in 
alternation requires additional hardware features for proper functioning, 

10 for example a second frame buffer for switching back and forth between 
the screens. Additionally, although this feature is not disclosed or 
apparently contemplated in Ouellette, using the Ouellette device in a 
wireless-transmission (e.g. RF) environment, for example in a wearable- 
computing context, would require e.g. twice as much information to 

15 wirelessly be transmitted and therefore would greatly increase the required 
bandwidth. 

SUMMARY OF THE INVENTION 
To overcome the above and other disadvantages, a screen 

20 peripheral system according to an embodiment of the invention includes a 
computing device for producing a main image and a touch-activated input 
device for generating and displaying a composite image visible to a user. 
The composite image simultaneously includes a representation of at least 
one key, for example a QWERTY keyboard, for activating an input 

25 function, and the main image provided by the computing device. The 
keyboard representation preferably is laid over the main image. 

The system implements variable-pixel control to form the 
keyboard representation and to form the main image, causing the pixels 
selected to form the keyboard representation and selected to form the main 

30 image to be dependent on each other. In other words, the keyboard- 
representation pixels are not independent of the main-image pixels. 
According to one embodiment, the main image is an output image 
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generated by an application being executed by the computing device. 

Various image-adjustment, mode-switching, zoom- 
up /zoom-down and other embodiments associated with invention are 
also described, as are corresponding methods. 
5 Embodiments of the invention have particular application to 

wearable computing devices, such as those available from ViA, Inc., 
Northfield, N4N. Attention also is directed to the following U.S. patents, 
each of which is incorporated by reference herein: 5,581,492; 5,572,401; 
"5355,490; 549T,65rand 5^85398, all of which are owned by ViA, Inc. 

10 

BRIEF DESCRIPTION OF THE DRAWINGS 
Embodiments of the invention will be described with respect 
to the figures, in which like reference numerals denote like elements, and 
in which: 

15 Figure 1 is a perspective view of a wearable computing 

environment according to an embodiment of the invention; 

Figure 2 shows a keyboard representation according to an 
embodiment of the invention; 

Figure 3 shows a main image according to an embodiment of 
20 the invention; 

Figure 4 shows a composite image according to an 
embodiment of the invention; 

Figure 5 shows a second composite image according to an 
embodiment of the invention; 
25 Figure 6 is a flow chart showing operational steps according to 

an embodiment of the invention; 

Figure 7 shows a reduced-key keyboard; 

Figure 8 shows a touch screen according to an embodiment of 
the invention; and 

30 Figures 9-13 show touch screens having a handle, according 

to embodiments of the invention. 
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DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 

Embodiments of the invention have wide application in a 
number of computing environments. The invention is particularly 
applicable to wearable computing environments, as discussed above, in 

5 which portability and compactness are fundamental considerations. The 
invention is also applicable to a number of different input/output devices. 
For example, embodiments of the invention contemplate i/o devices with 
full; partial, reduced-key, alphanumeric or non-alphanumeric keyboards 
comprising one or more "keys," "buttons," "contact zones" or the like. 

10 The invention also contemplates a number of input schemes in which a 
user manually or audibly directs certain keys to be activated, not just 
screens requiring direct manual contact with e.g. a user's finger. 
Proximity-based, pen-based and voice-based inputs are among the 
functionalities encompassed by embodiments of the invention. Therefore, 

15 although particular embodiments will be described with respect to touch 
screens, keyboards and touch input, the invention is intended to be 
broadly interpreted as needed and not necessarily Hmited to those 
particular embodiments. 

Additionally, various images can be combined/superimposed 

20 according to embodiments of the invention. In primary embodiments, the 
superimposed image is that of at least one key, e.g. a keyboard. The 
background image is that output by an application executed by a 
computing device, for example a graphics, spreadsheet, word-processing, 
etc. application for use in private, commercial, military, public-service 

25 fields or other fields. However, either the background image or the 
superimposed image can be of other t3^es; for example the background 
image, or both images, can be provided by a processor or storage device 
associated with the system, instead of by an application program per se. 
Either or both image can be the subject of an enlargement or reduction 

30 process, to provide composite images having elements of different sizes. 
More than two images can be combined according to the invention as well; 
any combination of application, keyboard, or other images can be 
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combined in a composite image. Therefore, although particular 
embodiments will be described herein with reference to application-output 
images, keyboards, etc., the invention is intended to be broadly interpreted 
as needed and not necessarily limited to those particular embodiments. 

5 Touch screen keyboard overlays according to embodiments of 

the invention can be selectively turned on and off by a user of an 
associated personal computing device, for example a wearable personal 
computer. A representation of a keyboard, as will be described, lies over a 
main image on the touch screen, the main image preferably being formed 

10 in cormection with a main software application. A keyboard overlay 
according to the invention eliminates a physical keyboard or the necessity 
of having a split screen with a touch keyboard on a lower portion and a 
main screen squeezed into the remaining upper portion, for example. A 
keyboard overlay according to the invention also reduces and/or 

15 eliminates the space-conflict problems that can arise when a superimposed 
keyboard occupies the same portion of the display screen as a critical 
portion of the application output image. 

Keyboard overlays according to the invention are especially 
advantageous for very small screens, because of the relatively large and 

20 accessible overlay "buttons" (keys) that are provided, along with the ability 
to see all of the data behind the keyboard at the same time. 

Embodiments of the invention are especially advantageous 
in situations where voice-based input is undesirable. Although voice- 
recognition technology is making significant advances, it still requires 

25 verbal input, which can be disruptive to a meeting, for example. Verbal 
input also potentially armounces to all within earshot of the speaker the 
exact notes, etc. that are being entered. According to embodiments of the 
invention, therefore, the user inputs directly to the machine in complete 
silence. Still, voice-based input may be desirable in many cases, so 

30 optional switching between voice input and keyboard-overlay input is also 
contemplated. This can be activated by a physical button or key on the side 
or other surface of the touchscreen or associated device, or by a virtual 
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button or key on the touch-sensitive portion of the screen itself, or by 
voice. 

Keyboard overlays according to embodiments of the 
invention can take a number of forms. For example, a solid-line or 

5 dashed/dotted-line overlay is contemplated, providing outlines of each 
key. Alternatively, and selectively as chosen by a user, each key can be 
represented by a dot, or as a letter-area zone between intervening markers 
such as x's, dots, etc. Ideally, the representation of the keyboard is provided 
faintly on the standard screen, so that it and data/images from the main 

10 program can be seen simultaneously. Variable-pixel controls, described in 
more detail below, can be provided to change the thickness, brightness, 
and /or dotted-ness of the keyboard representation. In many cases the 
keyboard overlay can be extremely faint. 

The characters of the keyboard themselves, e.g. the letters 

15 associated with each key or zone, can be turned off if the user has 
memorized the keyboard layout, for example. The keyboard can be active 
without being visible at all, for the user who can type effectively without 
even seeing representations of the keys/zones themselves. 

Contrast adjustment buttons are contemplated, preferably 

20 represented on the touch screen itself but possibly on a housing of the 
touch screen, to adjust contrast between the keyboard and the main screen. 
A keyboard on-off button is also contemplated. Alternatively, these 
features can be controlled by voice. 
Variabte-pjixel rontypls 

25 In accordance with embodiments of the invention, software- 

based variable-pixel controls are provided to determine and control which 
pixels of the touch screen will be used for displaying the keyboard 
representation and which pixels for displaying the main image. In some 
cases, each pixel of the screen is 100% dedicated to either the keyboard or 

30 the main image. In other cases, a touch screen pixels may be dedicated to 
both the keyboard and the main image, producing a "blended" effect as 
will be described. 
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Merging the image of the virtual keyboard with the normal 
output image on the display according to the invention can occur by a 
variety of methods and programming schemes. Using most current video 
graphics controllers, however, it is believed a highly efficient method 
5 available to perform the merging of images is to use bit-block or bit-block- 
type transfer operations, i.e. BitBlt operations. 

BitBlt operations provide an efficient method of performing 
logical combination of three sets of pixels on raster-based display devices. 
According to embodiments of the invention, BitBlt operations are used 
10 with the following three sets of pixels: 

(1) The original pixels on the display, i.e. the image that 
would be presented in the absence of the virtual 
keyboard. 

(2) The pixels representing the image of the virtual 
15 keyboard. 

(3) An image mask, allowing control of which pixels 
within the virtual keyboard will be merged with the 
original display pixels. 

Using BitBlt operations, the virtual keyboard can be 
20 combined with the display using a variety of effects. The following table 
summarizes the operations that can be performed: 
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The logical operations described herein each preferably have a 
different blending or merging effect. Although some of them do not have 
substantive effect, e.g. they just copy the source in to the destination or just 
fill it with zeros or ones and are ignorant of what the source and 

5 destination have, the large majority of these operations can be used to 
create a number of different effects to determine how to mesh the 
keyboard image and the application output image together. Additionally, 
the various logical operations can be combined as needed to produce an 
even greater variety of visual effects, e.g. harshness/intensity of one image 

10 with respect to the other. 

Two examples of the myriad meshing possibilities 
contemplated according to the invention will now be described. In the 
first example, the keyboard is blended with the original display 
application-output image by using 25% of the pixels to represent the 

15 keyboard and 75% of the pixels to represent the original display image. In 
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the second example, the two images are more truly blended together, with 
each pixel of the final composite display image receiving a contribution 
from both images. For the second example, the composite realized image 
can appear as e.g. a blended shadow from the original images. For both 

5 examples, it is assumed according to one embodiment that the graphics 
controller of the computer system is using an S-bit bit plane, which is 
typical of most personal computers supporting 256 colors per pixel. Of 
course, those of ordinary skill will readily be able to adapt these examples 
7or use in other current and future graphics controllers, bit planes, etc. 

10 In the first example, where the keyboard is blended with the 

display by using 25% of the pixels to represent the keyboard and 75% of the 
pixels to represent the original display image, each display update 
preferably is performed in three steps. The first step removes that portion 
of the keyboard image that is not to be seen, i.e. it eliminates 75% of the 

15 keyboard image. The second step removes that portion of the main 
display image that is not to be seen, i.e. it eliminates 25% of the main 
display image. The third step merges these two images. The following 
pseudocode represents one embodiment of the associated processing used 
according to this example: 

20 

reduced^kbd := BitBlt (source r:>keyboard image, 

destination =>keyboard image, 
mask =>[OxFO, 0x00, 0x00, 0x00, 

OxFO, 0x00, 0x00, 0x00, ...], 
operation =>merge copy) ; 
reduced^display := BitBlt (source => original display image, 

destination => original display image, 
mask =>(0x00, OxFO, OxFO, OxFO, 

0x00. OxFO, OxFO, OxFO, ...], 
operation => merge copy) ; 
display := BitBlt (source => reduced_kbd, 

destination => reduced.display , 

mask => null, 

operation => source paint); 
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With the first example, for performance reasons the first 
BitBlt can be performed once at initialization and then stored for reuse in 
future display updates. Essentially, the process eliminates from the two 
images those pixels that are not desired for the single, final composite 
5 image that is displayed to the user. In this way, neither image contains 
overlapping pixels. In the combined image, therefore, each pixel is 
contributed 100% by either the keyboard or the display, but 0% from the 
alternate. 

With the second example, the two images are logically 
10 combined using a single BitBlt, according to one embodiment. The 
following pseudocode represents one embodiment of the associated 
processing used according to this example: 



display := BitBlt (source => keyboard image, 

15 destination => original display image, 

mask =>null, 

operation => source paint) ; 



Current embodiments of the invention employ the following 
20 assumptions for performance reasons. It is contemplated that 
embodiments of the invention are for typically available operating 
systems, ones that support multi-tasking, with each application having its 
own thread of control, and a graphical user interface using windows as the 
visual representation of each task. Examples of such operating systems are 
25 Microsoft Windows 95, Microsoft Windows NT, SCO Unix, and Sun's 
Solaris operating system. Embodiments of the invention are also for 
typical video graphics controllers, ones that support bit-block transfer 
(BitBlt) operations and DMA between processor memory and video 
memory. Graphics controllers from Chips and Technologies, S3, 
30 NeoMagic, Trident, and other companies all support these operations, for 
example. 

Embodiments of the invention should be useable without 
necessarily relying on these assumptions, as hardware and software 
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technologies evolve, for example. Even currently, implementation in 
environments other than those described above is contemplated according 
to the invention, even though implementation in at least some of those 
environments may be more complex and subject to slower operation. 
5 Thus, according to embodiments of the invention, and as 

shown in e.g. Figures 1-5, screen peripheral system 5 according to an 
embodiment of the invention includes computing device 10 for e.g. 
executing a software application, and touch-activated input device 15 for 
generating and displaying a composite image visible to a user 20 of screen 

10 peripheral system 5. The composite image includes, according to one 
embodiment, a representation of at least one key for activating an input 
function, e.g. a keyboard-input function. The composite image also 
includes a main display image, for example the output image produced by 
the software application being executed by computing device 10. The 

15 representation of at least one key is laid over the main image produced by 
the software application. 

Figure 1 shows screen peripheral system 5 in a wearable- 
computing environment, which as described earlier is especially well- 
suited to embodiments of the invention. Embodiments of the invention 

20 can also be used in connection with a miniature, low-power, spread- 
spectrum "bodyLAN''-type system allowing selective control based on 
proximity to a user. With such a system, touch-activated input device 15 
can be in either wired or wireless communication with computing device 
10 and/or other devices /systems, for example LANs, WANs, etc. Wired 

25 and wireless communication embodiments are also contemplated for use 
with non-wearable computing devices as well. Fiber optic, electrical, 
infrared, an other data-transmission schemes between e.g. the disclosed 
input device and the wearable or other computer, or between any other 
elements of the system, are contemplated. Embodiments of the invention 

30 are especially useful with tablet-type touch screens and other small, readily 
portable/pocketable input/output devices. 

Embodiments of the invention also can be used in 
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connection with heads-up and /or body-worn display devices, e.g. dual- 
screen stereo displays. A keyboard overlay according to the invention, 
preferably a voice-activated and /or voice-controlled overlay, appears on 
the display, allowing the user to input data without looking down. 
5 Screen peripheral system 5 implements variable-pixel control 

to form the representation of at least one key and to form the output 
image, for example in the maimer described above. The variable-pixel 
control causes the pixels used to form the representation of at least one key 
and the pixels used to form the output image to be directly dependent on 

10 each other, in a manner that for example is also described above. In other 
words, unlike the Ouellette reference disclosed above, the pixels chosen to 
represent the images are not independent. 

The variable-pixel control can provide contrast adjustment 
between the representation of at least one key and the output image 

15 produced by the software application. The contrast adjustment can include 
changing at least one of the thickness, brightness, and dotted-ness of the 
representation of at least one key. 

As referenced above, according to one example computing 
device 5 eliminates X% of the pixels of a full representation of at least one 

20 key, X% representing that portion of the keyboard that is not to be seen in 
the composite image. In the first example desaibed above, X=75, although 
X can have a variety of different values to suit a particular display or 
purpose. Computing device 5 also eliminates (100-X)%, e.g. 25% as 
described above, of the pixels of a full application output image. Here, 

25 (100-X)7o represents that portion of the application output image that is 
not to be seen in the composite image. Computing device 5 merges the 
pixel-eliminated key representation with the pixel-eliminated main image 
to form the composite image. Alternatively and /or additionally, as in the 
second example described above, the composite image can include a 

30 blended shadow of the representation of at least one key and the main 
image. Computing device 5 can control the lightness/darkness of the 
blended shadow by controlling the blending of pixels of the at least one key 
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representation and the main image, according to one embodiment. By 
changing the darkness of the pixels of the composite image, for example 
the keyboard portion of the composite image, the keyboard can be made to 
appear to "bleed through" the other image. Thus, the keyboard "steals 
5 away" appropriate pixels from the main image, and /or shades them 
differently in a selected maimer, to produce the keyboard image. 

As will be appreciated by those of ordinary skill in the art 
upon reading this disclosure, methods corresponding to the above- 
described apparatus and systems are also contemplated according to the 
10 invention. 

Figures 2-5 illustrate sample images and composite screens 
according to embodiments of the invention. Figure 2 illustrates a 
representation of at least one key - in this example a full QWERTY 
keyboard with seyeral function /option keys/buttons. Figure 3 illustrates a 

15 main image generated by a computing device. Figure 3 should also be 
interpreted to cover the situation where the main image is an output 
image or images of an application or applications being executed by a 
processor or other element of the computing device. Figure 4 shows a 
composite image formed in accordance with Example 1 above, and Figure 

20 5 shows a composite image formed in accordance with Example 2 above. 

Figure 6 illustrates a flow chart according to an embodiment 
of the invention, identifying the generalized processing that occurs to 
allow a virtual keyboard to be overlayed upon a normal display. This 
processing maintains consistency of the virtual keyboard image with the 

25 rest of the information being displayed. 

After start step 100, a virtual keyboard image is created, at step 
105. The keyboard is drawn as an overlay on the current display in step 
110, and then an event is awaited at step 115. When a touch event occurs 
at step 120, the virtual key touched is determined at step 125 and the 

30 keyboard event is sent to the current task/window at step 130. The next 
event is then awaited at step 115. 

If there is no touch event at step 120, it is determined whether 
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a pertinent window has been redrawn, at step 135. If so, the area of screen 
. redrawn is determined at step 140, and then it is determined at step 145 
whether the area of screen redrawn overlaps the keyboard image. If not, 
the next event is simply awaited at step 115. If so, the keyboard is 
5 (re)drawn as an overlay on the current display at step 110. It is also 
determined whether a window is deleted at step 150, and whether a 
window is created, at step 155. If the answer is yes, steps 140, 145 and 110 or 
115 occur as described above. If the answer is no, the next event is (still) 
awaited at step 115. 

10 Mpde'switchmg 

According to one embodiment, if a representation of a key or 
button is held "down" for a set time, for example one second, a cursor 
appears at the point of the depressed key and the remainder of the 
keyboard (optionally) disappears. By dragging a finger along the touch 

15 screen, the user can move the cursor to a desired location. According to 
one embodiment, upon entering the cursor mode a number of new 
keys/buttons (or representations thereof, as above) appear on a side of the 
screen, for example, cut, paste, copy, delete and/or hold buttons. An 
appropriately placed mouse button is also contemplated on the screen, e.g. 

20 in a corner of the screen, to be accessed by a user's thumb. A convenient 
holding mechanism for the screen, allowing easy access by the user's 
thumb, is described below. 

In use, a user types on the keyboard overlay for a desired 
time, and then holds one finger down for a set period of time to exit the 

25 keyboard mode and enter the cursor mode. Selected text can be 
highlighted using the above-described mouse button in e.g. a comer or on 
a side of the touch screen, and then the cursor can be dragged to hover 
over a cut button or other desired button. Movement of the user's finger 
becomes equivalent to movement of the mouse when the finger is held 

30 down, and the touch button for the mouse is in the comer of the screen so 
that it is accessible by the thumb. Upon exiting the cursor mode, the user's 
fingers are again used for typing. 
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Rec<ucec|-ky embodiments 

According to one keyboard overlay embodiment, the number 
of keys of a virtual QWERTY keyboard is reduced to e.g. twelve, nine of 
which are letter keys and the other three of which are shift, delete and 
5 select keys, for example. Note Figure 7, showing one type of reduced-key 
keyboard 200. Each letter key represents multiple letters, for example three 
letters or letter combinations. A word-recogrution program discerns the 
word intended to be typed by hitting the letter keys, e.g. upon depressing 
the select" (or space) keys. Reduced-key keyboard patterns enhance 
10 miniaturization achievable according to the invention and provide other 
advantages. 

According to one embodiment of the invention, a reduced- 
key keyboard is incorporated as a virtual or physical touch layout 210 on a 
standard touch screen/slidepad moiise 220, as shown in Figure 8. A 

15 program monitors the touch screen 220 continually, detecting when a 
virtual option button 230 or physical option button 240 is selected to 
change the mode of operation from a standard touchpad mouse ("relative" 
mode) to a keyboard-input touchpad ("absolute'' mode). Upon entering 
the keyboard mode, a key pattern overlay (e.g. a letter keyboard, number 

20 pad, and /or individual keys) according to the invention appears on the 
touchscreen directly and /or on an associated screen. It is then detected 
which key of the keyboard the finger of the user is tapping on to generate 
letter/word or other input. Using a reduced-key keyboard in cormection 
with a standard touchpad mouse, with mode-switching as described above, 

25 provides significant miniaturization, speed, and other advantages. 

According to another embodiment, the virtual or physical 
select or other buttons 230, 240 can be depressed multiple times to enter 
and /or exit various submodes of operation. The first touch can cause a 
change from a relative mode to an absolute mode, and subsequent touches 

30 to absolute submodes such as alphabet, numeric, punctuation and/or 
other sub-modes. For example, if a user desires to input e.g. punctuation 
while in a standard relative mouse mode, the user taps the appropriate 
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button or buttons twice, first to enter the absolute mode and then to enter 
the punctuation input sub-mode, enters the appropriate pvmctuation, and 
then taps the button or buttons twice again to return to an absolute keypad 
mode. One or a plurality of buttons can be used to switch between relative 

5 and absolute modes, and, within e.g. the absolute mode, to choose a 
keyboard, number pad, punctuation input sub-mode or other oprion. 

It is also known in the art to reduce the size of a full keyboard, 
with alphabetic, numeric and function sub-keyboards, by superimposing 
the keys of the sub-keyboards on top of each other. According to one 

10 example, the alphabetic sub-keyboard is superimposed on the numeric and 
function sub-keyboards. For numeric input, the numbers are printed not 
at the top of each key but where multiple adjacent letter keys meet, at the 
interstices between the keys. Four "quarter" keys thus operate together to 
register the desired number, but each virtual number key is full-sized. 

15 Beneath the four letter keys associated with each number are sensing areas, 
i.e. points or elements or regions, with activation of a sufficient number of 
sensing areas by the user's finger registering the desired number. This 
technology reduces the overall size of the keyboard, for example to credit- 
card size. Examples of these types of key layouts are shown in U.S. Patent 

20 No. 5,612,690; among others. U.S. Patent No. 5,612,690 is incorporated 
herein by reference. 

Superimposing the keys of various sub-keyboards on top of 
each other, such as an alphabetic sub-keyboard superimposed on a 
numeric and /or function sub-keyboard, can be combined with the 

25 reduced-key concepts disclosed above with respect to Figures 7-8, to further 
reduce the size of a keyboard or keyboard overlay according to the 
invention. With the layout of the Figure 7 embodiment, for example, 
number and /or function keys would be placed at the physical or virtual 
"interstices" between the threeletter keys shown. The resulting composite 

30 image would then be superimposed over an application output in a 
manner described previously, for example. 
Zoom embodiments 
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According to other embodiments, zoom-up /zoom-down 
capabilities are incorporated through a touch screen or other touch- 
sensitive input device, for example in a manner compatible with the 
above-described technologies. Once an appropriate zoom mode is 
5 activated, e.g. by virtual or physical keys, contact of one, two, three or more 
of a user's fingers with the touch-sensitive surface causes activation of an 
appropriate number of sensing areas and/or "zones" thereon to control 
screen size and /or the amount of magnification displayed. According to 
one example embodiment, one-finger contact causes normal screen size 

10 and /or magnification, two-finger contact causes a zoom-up to double 
size /magnification, three-finger contact causes triple size /magnification, 
and so on. Zoom-down capabilities function can be configured similarly. 
Sequential step-up or step-down is not required; for example triple- 
size /magnification can be reduced to normal size /magnification merely 

15 with the touch of one finger, without entering a double- 
size/magnification mode. 

Thus, the user is spared the burden of going to a file menu, 
pulling down to the zoom feature and then selecting zoom-in and zoom- 
out functions using e.g. the mouse. A virtual, physical, or voice-activated 

20 zoom function select button is activated, and, simultaneously or 
thereafter, the touchscreen/slidepad itself determines how much of the 
touchscreen is covered at any one time by the user's fingers (or other 
pointing device), providing a specific screen size/ magnification related to 
the number of fingers being used. According to a more specific 

25 embodiment, appropriate zoom functionality occurs based on the number 
of sensing areas or contact zones activated on the touch-sensitive input 
device. 

These features have special relevance to portable, wearable 
computing, in which miniaturized screens are used, for example on the 
30 wrist, and generally are not full-sized. To use existing software, a user may 
want to see screen features in larger, zoomed-in format at certain times 
and, at other times, in a zoomed-out format in reference to how items 
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look on an overall page. 
Other hardware embodiments 

One problem with prior touch screens is that for miniature 
computers and miniature screens, the user's hand is often bigger than the 

5 screen itself. According to the embodiments of the invention, therefore, 
retractable handle 260 (Figs. 9-10) slides relative to touch screen 270 
between extended and retracted positions, as shown at 280, to provide a 
convenient holding mechanism at either the left or right side of touch 
screen 270. To shift handle 260 from the left side to the right side, or vice 

10 versa, pivot 285 is provided to swing the handle over the touch screen, as 
shown at 290. In the retracted position, the handle catches on the back side 
of the touch screen, permitting the pivoting motion. Upon traveling to or 
remaining in an extended position, a ball or similar projection snaps into a 
lateral groove or equivalent mechanism to prevent pivoting. Thus, 

15 embodiments of the invention are significantly advantageous for use by 
both left-handed and right-handed individuals, as shown in Figure 10, as 
well as those who need to use alternative hands for different tasks. 
Handle 260 can also be used as a carrying handle (Figure 11), as part of a 
necklace embodiment (Figure 12), or in a wristband embodiment (Figure 

20 13) using e.g. a VELCRO strap. Additionally, according to another 
embodiment, a double-sided touch screen is provided, with the capability 
of flipping the screen by voice or by touch to accormnodate left-handed or 
right-handed gripping of the touch screen by the handle without pivoting 
the handle from one side to the other. • 

25 As indicated earlier, embodiments of the invention have 

application in voice-recognition and /or hands-free embodiments. A 
microphone /speaker can be built in the face of the touch-sensitive input 
device, permitting a user to look at and talk to the device without having a 
separate microphone /speaker. Voice commands can be used to change 

30 turn keyboard overlays and /or voice-input modes on and off, and to 
change contrast, keyboard representation types, keyboard /cursor modes, 
etc. Voice commands also can be used to activate the touch screen itself, so 
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that it doesn't turn on by e.g. being depressed in a user's pocket, for 
example. 

It is contemplated that features disclosed in this application, 
as well as those described in the above applications incorporated by 
reference, can be mixed and matched to suit particular circumstances. 
Various other modifications and changes will be apparent to those of 
ordinary skill. 
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WHATISaAIMgPg: 



1 1. A screen peripheral system, comprising: 

2 a computing device for providing a main image; and 

3 a touch-activated input device for generating and displaying a 

4 composite image visible to a user of the screen peripheral system, the 

5 composite image simultaneously including: 

6 a representation of at least one key, the representation of at 

7 least one key activating an input fimction; and 

8 the main image provided by the computing device, the 

9 representation of at least one key being laid over the main image; 

10 wherein the screen peripheral system implements variable-pixel 



11 control to form the representation of at least one key and to form the main 

12 image, the variable-pixel control causing the pixels selected to form the 

13 representation of at least one key in the composite image to depend on the 

14 pixels selected to form the main image. 



1 2. The screen peripheral system of claim 1, wherein the 

2 variable-pixel control provides contrast adjustment between the 

3 representation of at least one key and the main image produced by the 

4 computing device. 

1 3. The screen peripheral system of claim 2, wherein the contrast 

2 adjustment includes changing at least one of the thickness, brightness, and 

3 dotted-ness of the representation of at least one key. 

1 4. The screen peripheral system of claim 1, wherein the 

2 representation of at least one key is a dotted-line representation. 

1 5. The screen peripheral system of claim 1, wherein the 

2 representation of at least one key includes a representation of a full 

3 keyboard. 



SUBSTITUTE SHEET (RULE 26) 



wo 98/09270 



PCrAJS97/14176 



-21- 

1 6. The screen peripheral system of claim 1, wherein the touch- 

2 activated input device operates in two modes, the first mode being a 

3 keyboard mode in which a user activates input functions by intermittently 

4 touching the input device, and the second mode being a cursor mode in 

5 which the user moves a cursor associated with the touch-activated input 

6 device, the second mode being entered from the first mode by 

7 continuously touching the input device for a set time. 



1 7. ~ 7 The~screen peripheral system of claim 1, wherein the 

2 computing device forms the composite image by merging the 

3 representation of at least one key with the main image. 

1 8. The screen peripheral system of claim 1, wherein the 

2 computing device merges the representation of at least one key with the 

3 main image by using a bit-block-type transfer operation. 

1 9. The screen peripheral system of claim 8, wherein the bit- 

2 block-type transfer operation performs logical combinations of three sets of 

3 pixels: 

4 (a) the pixels of the main image; 

5 (b) the pixels of the representation of at least one key; and 

6 (c) an image mask for controlling which pixels of the main 

7 image will be merged with which pixels of the representation of at least 

8 one key. 

1 10. The screen peripheral system of claim 1, wherein each pixel 

2 of the composite image is contributed 100% by either the pixels of the main 

3 image or the pixels of the representation of the at least one key. 

1 11. The screen peripheral system of claim 1: 

2 wherein the computing device eliminates X% of the pixels of a full 

3 representation of at least one key, X% representing that portion of the 
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4 keyboard that is not to be seen in the composite image; 

5 further wherein the computing device eliminates (100-X)% of the 

6 pixels of a full main image, (100-X)% representing that portion of the main 

7 image that is not to be seen in the composite image; 

8 further wherein the computing device merges the pixel-eliminated 

9 key representation with the pixel-eliminated main image to form the 
10 composite image. 

1 12. The screen peripheral system of claim I, wherein the 

2 composite image includes a blended shadow of the representation of at 

3 least one key and the main image. 

1 13. The screen peripheral system of claim 12, wherein the 

2 computing device controls the lightness/darkness of the blended shadow 

3 by controlling the blending of pixels of the at least one key representation 

4 and the main image. 

1 14. The screen peripheral system of claim 1, wherein the 

2 computing device executes a software application, the main image being 

3 an output image produced by the software application executed by the 

4 computing device. 



1 15. The screen peripheral system of claim 1, wherein the touch- 

2 activated input device provides zoom-up /zoom-down capability 

3 dependent on the number of the user's fingers in contact with the touch- 

4 activated input device. 



1 16. The screen peripheral system of claim 1, wherein the touch- 

2 activated input device allows switching between relative and absolute 

3 input modes depending on the number of times a user contacts a selected 

4 area of the touch-activated input device. 
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1 17. The screen peripheral system of claim 1, wherein the touch- 

2 activated input device allows switching between keyboard alphanumeric 

3 sets depending on the number of times a user contacts a selected area of 

4 the touch-activated input device. 

1 18. A wearable computing system comprising the screen 

2 peripheral system of claim 1. 

1 19. A method of superimposing a representation of at least one 

2 key over a main image provided by a computing device, the method 

3 comprising: 

4 (a) using variable-pixel control to form a representation of at 

5 least one key, the representation of at least one key activating an input 

6 function, and to form the main image, the variable-pixel control causing 

7 the pbcels selected to form the representation of at least one key to depend 

8 on the pixels selected to form the main image; and 

9 (b) generating and displaying a composite image visible to a 



10 user of the screen peripheral system, the composite image simultaneously 

11 including the representation of at least one key and the main image 

12 produced by the computing device, the representation of at least one key 

13 being superimposed on the main image. 



1 20. The method of claim 19, wherein step (b) includes the step of 

2 merging the representation of at least one key with the main image. 

1 21. The method of claim 20, wherein the merging step includes 

2 the step of using a bit-block-type transfer operation. 

1 22. The method of claim 21, wherein step (b) further includes 

2 combining three sets of pixels: 

3 (aa) the pixels of the main image; 

4 (bb) the pixels of the representation of at least one key; and 
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5 (cc) an image mask for controlling which pixels of the main image 

6 will be merged with which pixels of the representation of at least one key. 

1 23. The method of claim 21, wherein step (b) further includes: 

2 (c) eliminating X% of the pixels of a full representation of at least 

3 one key, X% representing that portion of the keyboard that is not to be seen 

4 in the composite image; 

5 (d) eliminating {100-X)% of the pixels of a full main image, (100-X)% 

6 representing that portion of the main image that is not to be seen in the 

7 composite image; and 

8 (e) merging the pixel-eliminated key representation with the pixel- 

9 eliminated main image to form the composite image. 

1 24. The method of claim 19, wherein the composite image 

2 includes a blended shadow of the representation of at least one key and the 

3 main image. 

1 25. The method of claim 24, wherein step (b) includes controlling 

2 the lightness/darkness of the blended shadow by controlling the blending 

3 of pixels of the at least one key representation and the main image. 

1 26. A method of generating a composite image from at least two 

2 base images, the method comprising: 

3 (a) displaying a first of the base images; 

4 (b) creating an image of a second of the base images; 

5 (c) displaying the second base image as an overlay on the 

6 first base image; 

7 (d) waiting for a display event; and 

8 (e) either taking no action or redrawing the second base 

9 image as an overlay on the first base image, depending on the type of 

10 display event. 
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1 27. The method of claim 27, wherein step (e) includes 

2 determining whether an area of display redrawn during the display event 

3 overlaps the second base image. 
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